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EXECUTIVE  SUMMARY 


OBJECTIVE 

This  technical  report  discusses  the  activities  of  the  Naval  Innovative  Science  and  Engineering 
(NISE)  project  entitled  Deep  Eearning  Online  Course,  executed  in  fiscal  year  2016  at  Space  and  Naval 
Warfare  Systems  Center  Pacific  (SSC  Pacific). 


RESULTS 

The  project  was  successful  in  training  a  large  group  of  scientists  and  engineers  in  the  topic  of  deep 
learning,  a  subfield  of  machine  learning  with  an  increasing  number  of  application  areas.  Some  difficulties 
were  encountered  throughout  the  course,  which  are  documented  herein. 


RECOMMENDATIONS 

More  courses  of  this  sort  should  be  held  at  SSC  Pacific  in  the  future.  The  lessons  learned  contained  in 
this  report  should  be  taken  into  consideration  when  planning  future  courses. 
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1.  INTRODUCTION 


CS231N  is  a  course  offered  at  Stanford  University  in  Winter  2015  and  again  in  Winter  2016  on  the 
topic  of  “Convolutional  Neural  Networks  for  Visual  Recognition.”  The  instructor  was  Prof.  Fei-Fei  Li, 
who  is  well  known  and  is  a  leader  in  the  computer  vision  community.  All  of  the  course  materials  were 
made  available  online  [3],  and  the  course  notes  and  programming  assignments  are  exceptionally  well  writ¬ 
ten  and  documented.  Because  of  this,  and  because  this  is  a  particularly  hot  topic  at  the  moment,  the  course 
became  popular  with  many  people  outside  of  Stanford.  There  is  a  Reddit  page  for  the  course,  and  a  com¬ 
puter  science  professor  at  the  Naval  Postgraduate  School  (NPS),  Prof.  Mathias  Kolsch,  used  the  materials 
to  teach  his  own  version  of  the  course  at  NPS. 

Meanwhile,  topics  relating  to  CS231N  have  become  of  greater  and  greater  interest  to  many  scientists 
and  engineers  at  Space  and  Naval  Warfare  Systems  Center  Pacific  (SSC  Pacific).  The  machine  learning 
communify  began  organizing  if  self  in  2012,  which  inspired  a  group  of  people  fo  sfudy  an  online  course 
from  Coursera.com  on  fhe  basics  of  machine  learning  [4].  This  group  mel  approximafely  fwice  a  monfh 
for  seven  monfhs  fo  discuss  fhe  course  maferial  and  how  if  applied  fo  projecfs  af  SSC  Pacific.  Befween 
fhis  course,  fhe  machine  learning  speaker  series  and  workshop^  [1],  and  a  general  increase  in  inferesf  in 
machine  learning,  a  large  number  of  scienlisfs  and  engineers  had  become  inferesfed  in  learning  more  abouf 
fhese  relafed  fopics. 

In  FY2016,  workforce  developmenf  funding  was  provided  by  fhe  Naval  Innovafive  Science  and  En¬ 
gineering  (NISE)  Program  fo  supporf  a  group  of  SSC  Pacific  scienfisfs  and  engineers  who  wished  fo  fake 
fhe  CS231N  course,  led  by  principle  invesfigafor  Kafie  Rainey  (Code  56220).  The  funds  covered  primar¬ 
ily  labor  for  fhe  parficipanfs  fo  sfudy  fhe  maferial  alongside  fheir  projecf  work.  This  reporf  documenfs 
fhe  acfivifies  of  fhe  course  along  wifh  some  lessons  learned. 


*The  machine  learning  series  and  workshop  were  funded  in  part  under  NISE  Workforce  Development  in  FY2014-15.  The 
workshop  was  held  in  March  2015. 
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2.  COURSE  STRUCTURE 


The  Stanford  course  was  given  in  14  lectures,  which  were  recorded  on  video  in  Winter  2016.  In  addi¬ 
tion  to  the  lecture  slides  and  videos^,  detailed  course  notes  were  made  available  online.  There  were  also 
three  homework  assignments  with  starter  Python  code  aimed  at  helping  the  student  learn  to  code  the  algo¬ 
rithms  taught  in  the  course. 

As  there  was  no  “expert”  in  this  subject  available  to  teach  the  course,  the  participants  at  SSC  Pacific 
set  out  to  study  the  material  individually  with  support  and  motivation  from  the  larger  group.  We  held  in- 
person  meetings  and  also  had  several  avenues  for  online  collaboration. 


2.1  IN-PERSON  MEETINGS 

The  kickoff  meeting  was  held  on  November  16,  2015.  It  was  attended  by  31  people,  including  several 
participants  from  the  Pacific  C4ISR  Department  (Code  H)  and  SSC  Atlantic  dialing  in. 

Six  “coding  parties”  were  held,  two  for  each  programming  assignment.  The  idea  behind  the  parties 
was  to  get  a  bunch  of  people  in  a  room  (or  online)  at  the  same  time  to  work  on  the  assignments  con¬ 
currently  so  that  they  could  ask  each  other  for  help  if  necessary.  Twenty-six  unique  participants  attended 
the  coding  parties,  which  were  all  held  in  the  Collaborative  Innovation  Lab  (CoIL)  in  Building  A33. 

Upon  finding  fhaf  more  people  were  focusing  on  fhe  course  notes  rafher  fhan  fhe  programming  assign- 
menfs,  we  began  a  series  of  discussion  groups  where  one  person  would  work  fhrough  some  of  fhe  fechnical 
defails  from  fhe  course  notes.  Seven  discussion  groups  were  held  wifh  25  unique  affendees. 


2.2  ONLINE  COMMUNICATIONS 

Regular  communications  with  the  course  participants  were  made  via  an  email  list,  and  were  always  cross- 
posted  to  the  project’ s  wiki  space  and  to  Fusion.  The  hashtag  #deeplearning  shared  interesting  links  and 
information  on  Fusion.  To  share  code,  we  used  Spork,  a  Space  and  Naval  Warfare  Systerms  Command 
(SPAWAR)  internal  code  repository,  and  similar  repositories  on  the  Internet  such  as  Bitbucket  and  GitHub. 

Additionally,  a  Slack^  team  was  established  to  foster  communications  outside  of  SPAWAR-internal 
channels,  which  was  useful  for  non-SPAWAR  participants  and  those  embedded  at  non-SPAWAR 
locations.  Slack  was  mostly  used  to  share  links  and  to  ask  and  answer  questions  about  the  course  material 
and  about  deep  learning  in  general.  Professor  Mathias  Kolsch,  who  taught  a  course  at  Naval  Postgraduate 
School  (NPS)  based  on  this  material,  joined  the  Slack  team  and  answered  technical  questions  that  some 
participants  asked.  This  interaction  wouldn’t  have  been  possible  without  some  sort  of  external  tool. 


2.3  ADDITIONAL  ACTIVITIES 

The  deep  learning  course  inspired  renewed  activity  among  the  Machine  Learning  Series.  At  least 
three^  Machine  Learning  Series  talks  hosted  in  FY16  were  organized  in  part  with  course  funding.  In  April 

^Several  months  after  Stanford’s  winter  quarter  completed,  and  mid-way  through  SPAWAR’s  course,  Stanford  removed  the 
course  videos  from  youtube  .  com  due  to  legal  concerns.  They  remain  hosted  unofficially  on  archive  .  org. 

^  Slack  is  a  commercial,  free-to-use,  cloud-based  collaboration  tool  for  teams.  It’s  essentially  a  glorified  chat  room. 

"'other  Machine  Learning  Series  talks  took  place  throughout  the  year,  but  they  were  not  organized  by  the  deep  learning  course 
organizers,  and  so  their  attendance  metrics  are  not  contained  herein.  Generally,  if  a  technical  talk  takes  place  outside  of  a  funded 
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Dean  Lee  (Code  54320)  talked  about  his  work  doing  predietive  analyties  on  V-22  aireraft  for  the  Comprehensive 
Automated  Maintenance  Environment  Optimized/Readiness  Integration  Center  (CAMEO/RIC)  projeet. 

In  May,  Don  Waagen  from  the  Army’s  Aviation  and  Missile  Researeh,  Development,  and  Engineering  Center 
(AMRDEC)  shared  thoughts  on  automatie  target  reeognition  algorithms  in  a  talk  entitled  “Cargo  Cults,  Rabbit 
Holes,  and  the  Most  Important  Questions.”  And  in  June,  Vu  Tran,  a  researeher  with  Booz  Allen  Hamilton, 
presented  his  work  on  applieations  of  eonvolutional  neural  networks  for  image  and  video. 

There  was  also  an  attempt  this  year  to  establish  a  “journal  elub”  at  whieh  partieipants  would  take  turns 
sharing  the  teehnieal  details  of  a  paper  that  they  found  interesting  or  relevant  to  their  work.  Due  to  the  time 
eommitment  required  for  the  presenter,  this  idea  has  been  slow  to  take  off.  Only  two  journal  elub  meetings 
were  held  in  EY 16,  but  hopefully  we  ean  piek  it  baek  up  next  year.  In  Mareh,  Ben  Migliori  (Code  56150) 
diseussed  two  papers  on  generative  visualization  and  training  of  deep  eonvolutional  neural  networks  [6,  7] 
and  how  they  apply  to  his  own  work.  In  May,  Justin  Mauger  (Code  56150)  shared  work  looking  at  imagery 
from  the  perspeetive  of  topology  [2]. 

Eunding  was  used  to  support  attendanee  of  five  eourse  partieipants — Katie  Rainey  (Code  56220), 

Alexander  Corelli  (Code  56220),  Mark  Bilinski  (Code  56150),  Chris  Barngrover  (Code  71710),  and  John 
Reeder  (Code  53624) — at  the  2016  IEEE  Computer  Vision  and  Pattern  Reeognition  (CVPR)  eonferenee 
in  Eas  Vegas,  NV,  in  late  June.  Two  other  eourse  partieipants — Miehael  Walton  (Code  56150),  Kris  Gib¬ 
son  (Code  56430)  and  Josh  Harguess  (Code  56220) — also  attended  on  other  projeet  funding.  The  atten¬ 
dees  all  gained  knowledge  to  support  their  eurrent  and  future  projeets,  but  in  September  we  held  a  CVPR 
roundtable  event  to  share  what  we  learned  with  the  rest  of  the  deep  learning  eommunity.  With  most  of  the 
CVPR  attendees  present,  we  shared  some  of  the  interesting  researeh  we  saw  presented  and  answered  ques¬ 
tions  about  the  eurrent  state  of  eomputer  vision  researeh. 


2.4  MANAGEMENT 

The  eourse  reeeived  $98,500  in  PY16  Naval  Innovative  Science  and  Engineering  (NISE)  Program 
funding.  The  bulk  of  that  was  spent  on  partieipant  labor,  with  some  also  spent  on  eonferenee  fees  and 
travel  for  CVPR.  Monthly  spending  was  traeked  and  reported  to  NISE  Program  Manager  Robin  Eaird 
through  monthly  status  reports.  Course  initiation  doeuments  were  also  provided,  ineluding  a  projeet 
management  plan,  an  exeeution  summary,  a  quad  ehart,  and  an  environ-mental  review.  A  mid-year  review 
presentation  was  delivered  in  April  2016.  Those  partieipants  who  eompleted  programming  assignments 
shared  their  eode  on  Spork  or  other  eode  repositories.  An  end-of-year  summary  was  delivered  in 
September,  and  a  poster  was  presented  at  a  NISE  poster  session.  All  management  doeuments,  and  links  to 
all  code  repositories  can  be  found  on  the  course  wiki  page  [5]. 


workforce  development  effort  like  the  deep  learning  course  or  the  Machine  Learning  Series  several  years  back,  no  one  will  bother 
to  take  attendance. 
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3.  IMPACT 


To  calculate  participation  metrics  for  the  course,  we  consider  39  ways  this  course  has  provided  to 
the  SPAWAR  community  to  engage  with  the  deep  learning  community.  These  events  include  the  course 
kickoff  meeting,  the  seven  discussion  groups,  the  six  coding  parties,  the  CVPR  roundtable,  and  six  deep 
learning-related  talks  held  this  year.  This  also  includes  non-events  such  as  whether  someone  charged  to  the 
project  during  a  given  month,  whether  they  posted  to  Spork  their  code  from  one  of  the  three  course  assign¬ 
ments,  whether  they  follow  the  #deepleaming  hashtag  on  Fusion,  whether  they  signed  up  for  the  course 
mailing  list,  and  whether  they  joined  the  course’s  Slack  team.  This  encompasses  66  total  participants  from 
SPAWAR,  plus  five  Naval  Research  Enterprise  Internship  Program  (NREIP)  or  Science  and  Engineering 
Apprentice  Program  (SEAP)  interns,  and  several  participants  from  NPS.  Of  course,  some  of  these  “events” 
indicate  only  very  passive  participation,  but  generally  the  more  events  a  person  participates  in  the  more  en¬ 
gaged  in  the  course  they  can  be  considered  to  be.  Eigure  1  charts  the  number  of  people  who  participated 
in  n  or  more  events  throughout  the  course.  Thirty-two  people  participated  in  three  or  more  events,  which 
includes  people  who  indicated  real  interest  but  perhaps  did  not  have  the  time  to  engage  fully  in  the  course. 
Twenty  people  participated  in  seven  or  more  events,  which  includes  anyone  who  charged  for  seven  months. 
Eifteen  people  participated  in  13  or  more  events,  which  roughly  captures  the  core  active  participants,  and 
10  extremely  engaged  people  participated  in  17  or  more  events.  Overall,  19  people  charged  labor  to  the 
course,  there  were  70  total  attendees  (25  unique)  to  the  discussion  groups,  and  there  were  59  total  atten¬ 
dees  (27  unique)  to  the  coding  parties.  Seventy-five  people  total  participated  in  the  38  events.  Notably,  56 
of  those  participants  did  so  without  funding. 


80 
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Figure  1 .  Number  of  people  who  participated  in  n  or  more  events. 
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4.  DISCUSSION 


4.1  BENEFITS 

Overall,  there  was  strong,  eonsistent  partieipation  in  the  eourse,  and  it  gave  good  benefits  to  the  S&T 
eommunity  for  the  eost.  A  eouple  dozen  engineers  and  seientists  now  have  a  greater  understanding  of 
an  important  eoneept  in  maehine  learning,  making  them  better  equipped  to  design  new  systems  for  the 
warfighter,  evaluate  eontraetors’  algorithms,  and  answer  sponsors  when  they  ask,  ’’did  you  try  deep  learn¬ 
ing  on  this  problem?”  This  eourse  has  strengthened  the  S&T  eommunity  and  introdueed  seientists  and  en¬ 
gineers  to  people  and  projeets  outside  of  their  respeetive  divisions.  It  familiarized  seientists  and  engineers 
with  a  teehnieal  area  that  is  beeoming  relevant  to  more  and  more  projeets,  and  supported  a  eulture  of  learn¬ 
ing  at  the  Center.  It  faeilitated  the  hosting  of  teehnieal  talks  of  interest  beyond  just  the  eourse  partieipants, 
and  demonstrated  to  employees  that  the  Center  values  their  professional  development.  The  events  of  the 
eourse  sparked  ideas  and  diseussion  about  potential  new  projeets,  and  provided  a  model  for  other  similar 
eourses  in  other  teehnieal  areas. 


4.2  CHALLENGES 

Course  Material.  CS231n  is  an  aetive  undergraduate  eourse  at  Stanford  University,  it  is  not  an  overview 
intended  for  a  broad  internet  audienee.  It  turns  out  that  the  eourse  material  is  very  teehnieally  ehalleng- 
ing.  Many  partieipants  found  it  diffieult  to  keep  paee  with  the  eourse  in  only  the  small  number  of  hours 
for  whieh  they  reeeived  funding.  The  programming  assignments  were  time  eonsuming  and  espeeially  hard 
for  people  without  mueh  experienee  eoding  in  Python.  Consequently,  many  people  gave  up  on  the  pro¬ 
gramming  and  foeused  instead  on  understanding  the  eourse  notes  and  watehing  the  video  leetures.  This 
impaeted  attendanee  at  eoding  parties  and  explains  the  laek  of  eode  eommitted  to  Spork. 

Coding  Parties.  Attendanee  at  the  eoding  parties  dropped  after  the  first  set,  due  to  several  faetors.  First  is 
the  faet  that  many  people  did  not  spend  a  lot  of  time  working  the  programming  assignments,  as  diseussed 
above.  Seeond  is  logistieal  issues  that  made  eoding  parties  not  eondueive  to  eollaboration.  CoIL  is  not 
properly  set  up  for  groups  to  eode  together.  Not  everyone  worked  on  the  eourse  on  a  laptop,  so  they  eould 
not  always  bring  their  work  with  them  to  the  parties.  During  some  of  the  parties  the  eomputers  in  CoIL 
were  not  able  to  get  on  the  internet,  so  the  assignments  eouldn’t  be  worked  loeally.  Generally  it  was  hard 
to  work  together  when  not  everyone  was  working  at  the  same  paee.  This  beeame  more  true  for  the  seeond 
and  third  assignments.  Interesting  diseussions  were  had  during  the  parties,  but  not  mueh  eoding  was  ae- 
eomplished. 

Long  Distance  Collaboration.  There  was  a  lot  of  enthusiasm  early  on  from  partieipants  not  loeal  to 
San  Diego,  ineluding  two  students  in  Hawaii  and  a  group  at  SSC  Atlantie.  But  it  was  too  logistieally  dif¬ 
fieult  to  use  DCS  for  the  eoding  parties  or  diseussion  groups,  and  the  Hawaii  students  in  partieular  found 
it  frustrating  that  they  weren’t  able  to  fully  partieipate.  The  Slaek  team  was  meant  to  bring  the  geographi- 
eally  distant  together,  and  while  it  was  used  somewhat  there  simply  wasn’t  enough  aetivity  to  keep  people 
engaged,  with  many  people  not  signing  up  for  the  team  in  the  first  plaee. 

Funding.  Originally,  funding  was  requested  for  12  hours  per  month  per  partieipant  over  7  months  (84 
hours  total),  but  our  low-ball  offer  was  aeeepted  instead  whieh  gave  some  partieipants  only  6  hours  (42 
hours  total).  This  was  not  enough.  Partieipants  found  it  diffieult  to  earve  only  a  small  number  of  hours 
away  from  their  regular  projeet  work.  Spreading  the  eourse  over  seven  months  made  it  hard  to  dedieate  the 
large  ehunks  of  time  neeessary  to  learn  this  diffieult  material.  Beeause  of  this,  despite  not  having  enough 
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money,  the  projeet  was  perpetually  underspent.  Many  partieipants  only  eompleted  part  of  the  eourse  mate¬ 
rial  in  the  time  allotted,  or  did  not  gain  as  mueh  detailed  understanding  as  they  eould  have. 

It’s  not  elear  what  the  right  solution  is  to  helping  the  workforee  learn  new  skills  in  our  projeet-funded 
environment.  Twenty  hours  per  week  for  two  months  might  allow  greater  dedieation,  but  few  people  would 
be  able  to  spare  that  kind  of  time  away  from  their  projeets.  Shortening  the  amount  of  time  would  make  it 
hard  for  partieipants  who  had  travel  or  other  eonfliets  arise.  Possibly,  the  solution  is  to  organize  smaller 
groups  who  eould  eommit  to  a  eondensed  sehedule. 

The  idea  behind  our  diseussion  groups  was  to  ease  the  study  burden  for  some  partieipants.  One  vol¬ 
unteer  would  study  the  material  from  one  leeture  and  present  it  to  the  group.  This  way  people  who  had 
not  found  time  to  keep  up  with  their  reading  eould  still  attend  and  learn  something.  The  diseussion  groups 
worked  out  relatively  well.  They  were  well  attended  and  fostered  lots  of  interesting  dialog  between  people 
who  had  familiarity  with  using  the  eoneepts  in  praetiee  and  those  who  were  eompletely  unfamiliar  with 
them.  Unfortunately,  we  ran  out  of  volunteers  to  lead  the  diseussion  groups. 
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5.  FEEDBACK 


This  section  contains  feedback  from  several  of  the  course  participants. 

Alan  Li,  Code  H56F0: 

Without  something  like  this,  I  would  have  never  tinkered  with  machine  learning.  I  think  this  is 
a  great  way  to  get  your  feet  wet  in  a  foreign  technical  subject  matter.  The  flexibility  is  a  great 
safety  net  for  those  of  us  with  other  assignments  that  could  have  hard  deadlines.  This  type 
of  training  is  better  than  the  typical  multi-day /one- week  training  classes  SPAWAR  normally 
puts  out  because  everyone  can  work  at  their  own  pace  and  take  a  deeper  dive  into  the  subject 
matter. 

Lance  Nakamoto,  Code  H56F0: 

I  would  participate  in  more  projects  like  this  if  the  opportunity  arose.  Running  through  a  col¬ 
lege  course  is  very  good  because  we  get  lectures,  structure,  etc.  The  problem  I  encountered 
was  being  remote,  I  was  not  able  to  participate  in  the  discussions/study  group  sessions.  Slack 
ended  up  not  being  used  much  by  people,  so  after  a  while  it  was  just  me  and  Alan  (Zhao)  in 
Hawaii,  but  we  had  conflicting  schedules  making  it  hard  to  collaborate. 

Steve  Hobbs,  Code  52390: 

The  team  learning  experience  was  terrific,  with  many  kinds  of  learning  going  on  and  good 
help  from  knowledgeable  colleagues.  I  did  not  do  everything  that  was  in  the  course  but  I  got 
out  of  it  what  I  wanted  most:  (a)  some  programming  experience  for  a  few  of  the  algorithms, 
(b)  to  learn  something  about  what  ’deep’  learning  is,  and  (c)  to  come  to  understand  ‘where  we 
should  be  going  from  here’  at  SSC  PAC. 
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